Visual Cognition While Listening to Stories

Different people have different perceptual patterns in response to language. Different language accents may influence visual cognition by increasing cognitive load when different objects are referenced in stories. The purpose of this study was to record differences in eye movement patterns while subjects listened to stories in two different accents, which served as the accent-induced stimulus load to increase cognitive processing. Using the eye tracker, eye movement patterns were recorded as subjects looked at an object array on a computer screen. My hypothesis that subjects would delay eye movement patterns to target objects while listening to stories in a foreign accent as opposed to an American accent because of increased cognitive load was supported with a main effect in accent type of stories. VISUAL COGNITION WHILE LISTENING TO STORIES 3 Human Gaze Control in Response to Different Language Accents While looking at a scene, people direct their eye movements toward targets that are relevant to their current task (Henderson, 2003). Top-down processes include prior knowledge, previous experiences with the scene, and goals and expectations. However, gaze control also involves bottom-up processes when eye movements are controlled by basic visual features such as color, size, and shape of an object. Both processes work together to direct eye movements throughout the scene. The visual information in the scene thus plays an important role in determining where people will fixate and how long they will look at a particular location (Henderson, 2003). However, there are many other factors that contribute to scene perception besides visual stimuli including auditory and kinesthetic stimuli. For example, Huettig and Altmann (2005) observed eye movements in response to language involve an overlap between conceptual information conveyed by individual spoken words and conceptual knowledge associated with visual objects. One of the earliest demonstrations of this phenomena involved short stories that subjects listened to while viewing a grid of nine simple line drawings (Cooper, 1974). While subjects listened to the story, their eyes fixated on certain objects that were related to certain key words in the story almost immediately after the word was uttered. For example, when subjects heard the word lion, they would immediately look at the picture of the lion. But is it always the case that listening to language will drive eye movements to related objects? To answer this question, Cooper’s (1974) study examined the extent to which the meaning of spoken language is able to continually control the locus of eye fixation. He observed that there were three types of fixation modes during scene perception in response to language: visual-aural interaction mode, free-scanning mode, and point-fixation mode. In the visual-aural VISUAL COGNITION WHILE LISTENING TO STORIES 4 mode, subject’s eyes fixated on targets related to words just after the words were uttered. However, in free-scanning mode, control of eye movements by the auditory stimulus was disengaged and subjects continually altered their gaze direction to words that were unrelated to the words being presented. Likewise, in point-fixation mode, subject’s eyes fixated on one spot independent of the meaning of the utterances that they were hearing. Cooper (1974) observed that people frequently switched fixation modes while listening to spoken language. Cooper suggested that the different fixation modes reflected differences in the distribution of attention between auditory and visual stimuli, but did not propose a mechanism by which those differences might occur. Nevertheless, these results are consistent with studies of gaze control that have been conducted since Cooper’s pioneering work. Doherty-Sneddon, Bruce, Bonner, Longbotham, and Doyle (2002) observed that when people face high cognitive loads, they avert their gaze at critical points within a task or interaction to avoid processing of unnecessary, distracting, or arousing visual cues from their environment or to focus on task related comprehension. Similarly, Glenberg Schroeder, and Robertson (1998) suggested that averting eye gaze helps people to disengage from environmental stimulation, thereby enhancing the efficiency of internally directed cognitive processing. These results are consistent with the point-fixation mode of processing. Ehrlichman and Micic (2012) explained that eye fixations wander off from targets when engaging in high cognitive load tasks. Tasks that involved minimal long-term memory search and required focus on material in working memory produced the lowest eye movement rates, whereas tasks with high requirements for long-term-memory search produced the highest eye movement rates. They observed that people make multiple eye movements unrelated to the VISUAL COGNITION WHILE LISTENING TO STORIES 5 visual environment when they scan their surroundings for information in long-term memory and very few eye movements when they focus on information in the working memory buffer. In addition to manipulation of auditory stimulus, Clarke and Garrett (2004) observed that native speakers tend to have difficulty processing language when listening to a foreign accent due to different pronunciation of words and syllable structure, but soon become accustomed to the foreign accent. They explained that accented speech can slow perceptual processing due to poor word identification, which relates to the purpose of foreign accents as the perceptual load in my research. When people process different accents, they might use the visual world to search for surrounding information in their environment to help them understand the accent, thus, delaying fixation onset to target objects. Adank, Hagoort and Bekkerring (2010) have shown in their study that performance goes down when listening to foreign accents. They observed that people perform better in comprehension tasks when they imitate the designated foreign accent. Imitation may help subjects understand how non-native speakers pronounce words and syllables to better comprehend the foreign language to accomplish their designated tasks in the experiment. Likewise, if subjects imitate the accented speech in their minds, perhaps this will affect their eye movements. This experiment examined whether cognitive load plays a role in the distribution of attention between auditory and visual stimuli by manipulating the load due to either auditory, visual, or extraneous tasks. This experiment attempted to replicate and extend the Cooper (1974) study by manipulating the auditory aspect of the experiment by instructing subjects to listen to stories in different accents of the English language. Specifically, I examined whether the time to fixate on language-related objects on a screen in response to target words in a story increases as VISUAL COGNITION WHILE LISTENING TO STORIES 6 cognitive load is increased (due to shifts to point-fixation or free-fixation). I measured when subjects’ eyes fixated on specific targets and where they looked when not focusing on the targets. When subjects engage in high cognitive load that includes multiple tasks, they may stop doing one task to better focus on another task, which may lead their eyes to look away from their immediate environment. Based on Cooper’s (1974) study, working memory load would be expected to influence point fixations, but would perceptual load such as foreign accents have the same effect? Do different types of cognitive loads lead to different off-task behaviors? The purpose of this study was to record differences in eye movement patterns while subjects listened to stories in two different accents, which served as the accent-induced stimulus load to increase cognitive processing. In my study, I hoped to find whether different loads lead to more or less of task behavior and try to identify what kind of off-task behavior is occurring.